System and method of developing and managing a training program

ABSTRACT

A method of developing and managing a training program is provided. The method includes defining the training program from a plurality of parameters, defining a desired outcome of the training program for at least one learner, and determining a predicted outcome for the training program for the at least one learner. The predicted outcome is determined based on actual performance data for the plurality of parameters for the at least one learner. The method also includes adjusting the plurality of parameters of the training program if a difference between the desired outcome and the predicted outcome is greater than a predetermined threshold.

BACKGROUND

The field of the present disclosure relates generally to adaptivelearning techniques and, more specifically, to a learning effectivenessadjustment and optimization methodology that properly challengeslearners based on the principles of predictive analytics andprobabilities.

Training and education is typically conducted in the same way for alllearners, which results in dissatisfaction and demotivation for bothhigh-performing learners and low-performing learners. For example, whenthe same training and education technique is utilized for all learnersin a group of learners, high-performing learners are demotivated as aresult of not being properly challenged, and low-performing learners aredemotivated as a result of a mental condition known as “learnedhelplessness.”

At least some known training and education techniques are adaptable byeither manually adjusting the parameters of an examination orsimulation, or by automatically adjusting the parameters based on thepast performance of an individual learner. However, manual adjustmentsare influenced by inherent cognitive biases that effect a person'sinterpretation of information and probabilities. For example, studentsconducting self-study may have a tendency to over-study and focus ontopics that have already been mastered. Moreover, automatic adjustmentstypically only modify a lesson at a high level by forcing a learner tore-attempt a particular lesson or by providing the option to skipadditional lessons on topics that have already been mastered.

BRIEF DESCRIPTION

In one aspect, a method of developing and managing a training program isprovided. The method includes defining the training program from aplurality of parameters, defining a desired outcome of the trainingprogram for at least one learner, and determining a predicted outcomefor the training program for the at least one learner. The predictedoutcome is determined based on actual performance data for the pluralityof parameters for the at least one learner. The method also includesadjusting the plurality of parameters of the training program if adifference between the desired outcome and the predicted outcome isgreater than a predetermined threshold.

In another aspect, a system for use in developing and managing atraining program is provided. The system includes a user interface, anda computing device coupled in communication with the user interface. Thecomputing device is configured to define the training program from aplurality of parameters, define a desired outcome of the trainingprogram for at least one learner, and determine a predicted outcome forthe training program for the at least one learner. The predicted outcomeis determined based on actual performance data for the plurality ofparameters for the at least one learner. The method is also configuredto receive, via the user interface, an adjustment of the plurality ofparameters if a difference between the desired outcome and the predictedoutcome is greater than a predetermined threshold.

In yet another aspect, a computer-readable storage media havingcomputer-executable instructions embodied thereon for use in developingand managing a training program is provided. When executed by at leastone processor, the computer-executable instructions cause the processorto define the training program from a plurality of parameters, define adesired outcome of the training program for at least one learner, anddetermine a predicted outcome for the training program for the at leastone learner. The predicted outcome is determined based on actualperformance data for the plurality of parameters for the at least onelearner. The computer-executable instructions also cause the processorto adjust the plurality of parameters of the training program if adifference between the desired outcome and the predicted outcome isgreater than a predetermined threshold.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of an exemplary system for use indeveloping a training program.

FIG. 2 is an exemplary user interface that may be used to assess thelearning effectiveness of a flight simulation program.

DETAILED DESCRIPTION

The implementations described herein relate to systems and methods thatprovide a customizable reporting and optimization tool to facilitatedeveloping, planning, and managing a training program to enhance thelearning effect for an individual learner or a group of learners. Morespecifically, the systems and methods described herein enable parametersof the training program to be displayed and individually adjusted suchthat an instructor is able to view the predicted learning effect ofadjusting the parameters. The predicted learning effect is based onactual performance data for the learner, and the parameters of thetraining program are adjusted to ensure the learner is properlymotivated. For example, the parameters are adjusted such that thelearner is neither overwhelmed nor underwhelmed by the training programto facilitate reducing apathy and anxiety in the learner. As such, thelearner is properly motivated, and an efficient and effective learningenvironment is provided.

Described herein are computer systems that facilitate adjusting atraining program. As described herein, all such computer systems includea processor and a memory. However, any processor in a computer devicereferred to herein may also refer to one or more processors wherein theprocessor may be in one computing device or a plurality of computingdevices acting in parallel. Additionally, any memory in a computerdevice referred to herein may also refer to one or more memories whereinthe memories may be in one computing device or a plurality of computingdevices acting in parallel.

As used herein, a processor may include any programmable systemincluding systems using micro-controllers, reduced instruction setcircuits (RISC), application specific integrated circuits (ASICs), logiccircuits, and any other circuit or processor capable of executing thefunctions described herein. The above examples are example only, and arethus not intended to limit in any way the definition and/or meaning ofthe term “processor.”

As used herein, the term “database” may refer to either a body of data,a relational database management system (RDBMS), or to both. As usedherein, a database may include any collection of data includinghierarchical databases, relational databases, flat file databases,object-relational databases, object oriented databases, and any otherstructured collection of records or data that is stored in a computersystem. The above examples are for example purposes only, and thus arenot intended to limit in any way the definition and/or meaning of theterm database. Examples of RDBMS's include, but are not limited toincluding, Oracle® Database, MySQL, IBM® DB2, Microsoft® SQL Server,Sybase®, and PostgreSQL. However, any database may be used that enablesthe systems and methods described herein. (Oracle is a registeredtrademark of Oracle Corporation, Redwood Shores, Calif.; IBM is aregistered trademark of International Business Machines Corporation,Armonk, N.Y.; Microsoft is a registered trademark of MicrosoftCorporation, Redmond, Wash.; and Sybase is a registered trademark ofSybase, Dublin, Calif.)

In one implementation, a computer program is provided, and the programis embodied on a computer readable medium. In an example implementation,the system is executed on a single computer system, without requiring aconnection to a sever computer. In a further implementation, the systemis being run in a Windows® environment (Windows is a registeredtrademark of Microsoft Corporation, Redmond, Wash.). In yet anotherimplementation, the system is run on a mainframe environment and a UNIX®server environment (UNIX is a registered trademark of X/Open CompanyLimited located in Reading, Berkshire, United Kingdom). The applicationis flexible and designed to run in various different environmentswithout compromising any major functionality. In some implementations,the system includes multiple components distributed among a plurality ofcomputing devices. One or more components may be in the form ofcomputer-executable instructions embodied in a computer-readable medium.

As used herein, an element or step recited in the singular and proceededwith the word “a” or “an” should be understood as not excluding pluralelements or steps, unless such exclusion is explicitly recited.Furthermore, references to “exemplary implementation” or “oneimplementation” of the present disclosure are not intended to beinterpreted as excluding the existence of additional implementationsthat also incorporate the recited features.

As used herein, the terms “software” and “firmware” are interchangeable,and include any computer program stored in memory for execution by aprocessor, including RAM memory, ROM memory, EPROM memory, EEPROMmemory, and non-volatile RAM (NVRAM) memory. The above memory types areexample only, and are thus not limiting as to the types of memory usablefor storage of a computer program.

Referring to the drawings, FIG. 1 is a schematic illustration of anexemplary system 100 for use in developing a training program. In theexemplary implementation, system 100 is used to define a trainingprogram from a plurality of parameters, define a desired outcome of thetraining program for at least one learner, determine a predicted outcomefor the training program for the at least one learner, and receive, viaa user interface, an adjustment of the plurality of parameters if adifference between the desired outcome and the predicted outcome isgreater than a predetermined threshold. System 100 includes a trainingprogram management server 102, a database server 104 within trainingprogram management server 102, and a database 106 coupled incommunication with database server 104. Training program managementserver 102 is a computing device that facilitates managing developmentand administration of the training program, as will be described in moredetail below.

As used herein, “training program” refers to any singular trainingevent, an aggregation of training events, or a portion of a trainingevent. As such, system 100 may be used to develop, plan, and manage aportion of a simulation event, a complete simulation event, or a seriesof events (e.g., a class or course).

In the exemplary implementation, system 100 also includes a plurality ofuser interfaces, such as a first user interface 108, a second userinterface 110, and a third user interface 112. The plurality of userinterfaces are computers that include a web browser or a softwareapplication, which enables first user interface 108, second userinterface 110, and third user interface 112 to access training programmanagement server 102 using the Internet, a local area network (LAN), ora wide area network (WAN). More specifically, the plurality of userinterfaces are communicatively coupled to the Internet through one ormore interfaces including, but not limited to, at least one of anetwork, such as the Internet, a LAN, a WAN, or an integrated servicesdigital network (ISDN), a dial-up-connection, a digital subscriber line(DSL), a cellular phone connection, a satellite connection, and a cablemodem. The plurality of user interfaces can be any device capable ofaccessing the Internet including, but not limited to, a desktopcomputer, a laptop computer, a personal digital assistant (PDA), acellular phone, a smartphone, a tablet, a phablet, or other web-basedconnectable equipment. In some implementations, one or more of the userinterfaces are either a dedicated instructor interface or a dedicatedlearner interface. Alternatively, system 100 includes a single userinterface that acts as both an instructor interface and a learnerinterface.

Database server 104 is coupled in communication with database 106 thatstores data. Database 106 stores data thereon, such as parameters fordefining the training program. For example, database 106 storesparameters of a flight simulation and/or parameters of an examinationthereon. The parameters of the flight simulation include, but are notlimited to, at least one of weather conditions, technologicalmalfunctions, types of aircraft, and third party interference. Theparameters of the examination include, but are not limited to, at leastone of a duration of the examination, types of questions on theexamination, and a number of each type of question on the examination.Moreover, database 106 stores actual performance data for at least onelearner thereon. The actual performance data includes recent performancedata and older performance data for the at least one learner. In analternative implementation, database 106 also stores empiricalperformance data thereon. Empirical performance is generally defined bythe effect of well-known variables or criteria on performance data for atheoretical learner. For example, it may be well-known that it is moredifficult to land an aircraft in windy conditions, as opposed to calmconditions.

In the exemplary implementation, database 106 is stored remotely fromtraining program management server 102. Alternatively, database 106 isdecentralized. Moreover, as will be described in more detail below, auser can access training program management server 102 or database 106via one or more of first, second, and third user interfaces 108, 110,and 112. Training program management server 102 is coupled incommunication with the first, second, and third user interfaces 108,110, and 112. In some implementations, training program managementserver 102 is decentralized and includes of a plurality of computerdevices which work together as described herein.

In operation, an instructor 114 interacts with first user interface 108to access a training program development/management tool on trainingprogram management server 102. As described above, training programmanagement server 102 is a computing device that facilitates managingdevelopment and administration of the training program. Morespecifically, training program management server 102 defines thetraining program from a plurality of parameters and defines a desiredoutcome of the training program for at least one learner. The parametersof the training program and the desired outcome may be selected at firstuser interface 108 by instructor 114 to dynamically tailor and/orenhance the learning effectiveness of the training program. Trainingprogram management server 102 further determines a predicted outcome forthe training program for the at least one learner. The predicted outcomeis determined based on actual performance data for the plurality ofparameters for the at least one learner.

As described above, the learning effectiveness of a training program isbased at least partially on a level of difficulty of the trainingprogram such that the learner is neither overwhelmed nor underwhelmed.As such, in the exemplary implementation, training program managementserver 102 defines the desired outcome having a quantifiable achievementlevel of less than 100 percent. Put another way, the desired outcome isselected such that the at least one learner fails to achieve a perfectscore, and thus, feels properly challenged. Additionally, the desiredoutcome is selected such that the at least one learner is not overlychallenged and demotivated. As such, in one implementation, trainingprogram management server 102 defines the desired outcome having thequantifiable achievement level defined within a range between about 60percent and about 80 percent, depending on other motivating factors.

In some implementations, training program management server 102 displaysthe predicted outcome for the training program for the at least onelearner on first user interface 108. Instructor 114 is then able todetermine if the parameters of the training program will properlychallenge the at least one learner once the training program isadministered. For example, system 100 provides informs instructor 114 ofan imbalance between the desired outcome and the predicted outcome and,if a difference between the desired outcome and the predicted outcome isgreater than a predetermined threshold, system 100 enables instructor114 to adjust the parameters of the training program. More specifically,training program management server 102 receives, via first userinterface 108, an adjustment of the plurality of parameters if thedifference between the desired outcome and the predicted outcome isgreater than the predetermined threshold. Similar to the selection ofthe desired outcome, the adjustment to the plurality of parameters issuch that the predicted outcome has a quantifiable achievement level ofless than 100 percent. As such, receiving the adjustment of theplurality of parameters ensures an effective learning environment isprovided to the at least one learner.

Moreover, as described above, actual performance data for the at leastone learner is stored on database 106. In the exemplary implementation,the actual performance data includes recent performance data and olderperformance data for the least one learner. As such, in oneimplementation, training program management server 102 accesses theactual performance data that includes recent performance data and olderperformance data for the at least one learner, and determines thepredicted outcome for the training program for the at least one learnerbased on a weighted average of the actual performance data. The weightedaverage has a greater emphasis on the recent performance data than theolder performance data. As such, the predicted outcome calculated bytraining program management server 102 is more accurately determined.

Once the training program has been developed and the predicted outcomesubstantially coincides with the desired outcome, the training programis then administered to the at least one learner on second userinterface 110 and third user interface 112. Moreover, coupling first,second, and third user interfaces 108, 110, and 112 in communicationwith each other via training program management server 102 enablesinstructor 114 to tailor the training program administered to a group oflearners quickly based on the actual performance data for each learner.

Mathematics Examination

In the example articulated below, an instructor uses the systems andmethods described herein to develop a learning activity for a group oflearners. More specifically, the learning activity is a mathematicsexamination that includes a set of addition problems and a set ofsubtraction problems to be solved in a certain time period. Theparameters of the mathematics examination are the number of additionproblems, the number of subtraction problems, and the duration forcompleting the examination.

Referring to Table 1, actual performance data for a group of learners isdisplayed. Learner A is very good at addition (95%), but is not as goodat subtraction (70% at the same pace). Learner B is not good at addition(40%-80% depending on duration), but is very good at subtraction (90%+).Learner C is good at both addition and subtraction (90%), but needs moretime than both Learner A and Learner B to achieve a high performancelevel (60%).

TABLE 1 Actual Performance Data Average Addition Score AverageSubtraction Score Learner and Answer Rate and Answer Rate A 95% @ 2questions/min 70% @ 2 questions/min 90% @ 1 question/min B 40% @ 2questions/min 90% @ 2 questions/min 75% @ 1 question/min 95% @ 1question/min 80% @ 0.5 questions/min C 60% @ 2 questions/min 60% @ 2questions/min 90% @ 1 question/min 90% @ 1 question/min

TABLE 2 Report of Predicted Outcomes Parameters 10 Addition 10 Addition8 Addition 15 Addition 10 Subtraction 10 Subtraction 12 Subtraction 5Subtraction Learner 10 Minutes 20 Minutes 10 Minutes 10 Minutes A(.95*10 + (.95*10 + (.95*8 + (.95*15 + .70*10)/ .90*10)/ .70*12)/.70*5)/ 20 = 0.83 20 = 0.93 20 = 0.80 20 = 0.89 B (.40*10 + (.75*10 +(.40*8 + (.40*15 + .90*10)/ .95*10)/ .90*12)/ .90*5)/ 20 = 0.65 20 =0.85 20 = 0.70 20 = 0.53 C (.60*10 + (.90*10 + (.60*8 + (.60*15 +.60*10)/ .90*10)/ .60*12)/ .60*5)/ 20 = 0.60 20 = 0.90 20 = 0.60 20 =0.60

As shown in Table 2, the parameters of the examination were adjusted andthe predicted outcome for each Learner was determined. As describedabove, the predicted outcome for each Learner should have a quantifiableachievement level of less than 100 percent to facilitate reducing apathyin the Learners. More specifically, the best learning typically occurswhen the predicted outcome for a training program has a quantifiableachievement level defined within a range between about 60 percent andabout 80 percent. As such, in the above example, the examination thatincluded 8 addition problems, 12 subtraction problems, and a duration of10 minutes had a combination of parameters that would result in the mosteffective learning experience for Learners A, B, and C. In analternative implementation, parameters of the examination can beindividually tailored for each Learner such that Learners A, B, and Ceach take a different examination.

Flight Simulation

Referring to FIG. 2, results and performance data are displayed for aflight simulation training program that includes a landing approachlesson. Displayed on user interface 108 are a parameter selection window116, a class performance window 118, and an individual performancewindow 120. Parameter selection window 116 enables an instructor toadjust the parameters of the flight simulation training program, such asweather conditions including rain, cross-wind, and wind shear, andtechnological malfunctions including engine malfunctions, controlmalfunctions, and indicator malfunctions. Class performance window 118displays each student's predicted performance level when differentparameters are selected for the flight simulation program. The predictedperformance level for the students is rated on a challenge level scaleand a skill level scale, each from low to high. A student whosepredicted performance level falls within the “Apathy” portion of thescale and at low levels of both the challenge level scale and the skilllevel scale may generally be classified as having a quantifiableachievement level of about 100 percent. As such, the parameters of theflight simulation training program may need to be adjusted to ensure astudent is properly challenged.

In the exemplary implementation, moderate levels of rain and cross-windare selected for the flight simulation training program. The predictedperformance levels are determined based on performance data for eachstudent for a flight simulation having similar parameters. In theexemplary implementation, Student A would be “in the flow”, Student Bwould be “aroused”, and Student C would be on the relaxed side offeeling “in control”.

Individual performance window 120 displays a student's predictedperformance level in response to each selected parameter for the flightsimulation training program. In the exemplary implementation, it isshown that Student A finds “rain” to be boring and at the low end of thechallenge level scale, but “cross-wind” is at the correct challengelevel. As such, the instructor is provided with a tool that enableshim/her to view and dynamically adjust the parameters of a flightsimulation training program to ensure the challenge level and skilllevel is properly determined for each student.

This written description uses examples to disclose variousimplementations, including the best mode, and also to enable any personskilled in the art to practice the various implementations, includingmaking and using any devices or systems and performing any incorporatedmethods. The patentable scope of the disclosure is defined by theclaims, and may include other examples that occur to those skilled inthe art. Such other examples are intended to be within the scope of theclaims if they have structural elements that do not differ from theliteral language of the claims, or if they include equivalent structuralelements with insubstantial differences from the literal language of theclaims.

What is claimed is:
 1. A method of developing and managing a trainingprogram, said method comprising: defining the training program from aplurality of parameters; defining a desired outcome of the trainingprogram for at least one learner; determining a predicted outcome forthe training program for the at least one learner, the predicted outcomedetermined based on actual performance data for the plurality ofparameters for the at least one learner; and adjusting the plurality ofparameters of the training program if a difference between the desiredoutcome and the predicted outcome is greater than a predeterminedthreshold.
 2. The method in accordance with claim 1, wherein defining adesired outcome comprises defining the desired outcome having aquantifiable achievement level of less than 100 percent.
 3. The methodin accordance with claim 2, wherein defining a desired outcome comprisesdefining the desired outcome having the quantifiable achievement leveldefined within a range between about 60 percent and about 80 percent. 4.The method in accordance with claim 1, wherein adjusting the pluralityof parameters comprises adjusting the plurality of parameters such thatthe predicted outcome has a quantifiable achievement level of less than100 percent.
 5. The method in accordance with claim 1, wherein theactual performance data comprises recent performance data and olderperformance data for the at least one learner, wherein determining apredicted outcome comprises determining the predicted outcome based on aweighted average of the actual performance data, the weighted averagehaving a greater emphasis on the recent performance data than the olderperformance data.
 6. The method in accordance with claim 1, whereindefining the training program comprises defining parameters of a flightsimulation including at least one of weather conditions, technologicalmalfunctions, types of aircraft, and third party interference.
 7. Themethod in accordance with claim 1, wherein defining the training programcomprises defining parameters of an examination including at least oneof a duration of the examination, types of questions on the examination,and a number of each type of question on the examination.
 8. The methodin accordance with claim 1, wherein defining a desired outcome comprisesdefining a quantifiable achievement level threshold for the desiredoutcome for a plurality of learners, said method further comprising:adjusting the plurality of parameters for the training program such thata predicted outcome for each learner of the plurality of learners fallswithin the quantifiable achievement level threshold.
 9. A system for usein developing and managing a training program, said system comprising: auser interface; and a computing device coupled in communication withsaid user interface, said computing device configured to: define thetraining program from a plurality of parameters; define a desiredoutcome of the training program for at least one learner; determine apredicted outcome for the training program for the at least one learner,the predicted outcome determined based on actual performance data forthe plurality of parameters for the at least one learner; and receive,via said user interface, an adjustment of the plurality of parameters ifa difference between the desired outcome and the predicted outcome isgreater than a predetermined threshold.
 10. The system in accordancewith claim 9, wherein said computing device is further configured todefine the desired outcome having a quantifiable achievement level ofless than 100 percent.
 11. The system in accordance with claim 10,wherein said computing device is further configured to define thedesired outcome having the quantifiable achievement level within a rangebetween about 60 percent and about 80 percent.
 12. The system inaccordance with claim 10, wherein said computing device is furtherconfigured to receive, via said user interface, a selection of thedesired outcome for the at least one learner.
 13. The system inaccordance with claim 9, wherein said computing device is furtherconfigured to receive, via said user interface, the adjustment of theplurality of parameters such that the predicted outcome has aquantifiable achievement level of less than 100 percent.
 14. The systemin accordance with claim 9, wherein said computing device is furtherconfigured to: access actual performance data that includes recentperformance data and older performance data for the at least onelearner; and determine the predicted outcome for the training programfor the at least one learner based on a weighted average of the actualperformance data, the weighted average having a greater emphasis on therecent performance data than the older performance data.
 15. Acomputer-readable storage media having computer-executable instructionsembodied thereon for use in developing and managing a training program,wherein, when executed by at least one processor, thecomputer-executable instructions cause the processor to: define thetraining program from a plurality of parameters; define a desiredoutcome of the training program for at least one learner; determine apredicted outcome for the training program for the at least one learner,the predicted outcome determined based on actual performance data forthe plurality of parameters for the at least one learner; and adjust theplurality of parameters of the training program if a difference betweenthe desired outcome and the predicted outcome is greater than apredetermined threshold.
 16. The computer-readable storage media inaccordance with claim 15, wherein the computer-executable instructionsfurther cause the processor to define the desired outcome having aquantifiable achievement level of less than 100 percent.
 17. Thecomputer-readable storage media in accordance with claim 16, wherein thecomputer-executable instructions further cause the processor to definethe desired outcome having the quantifiable achievement level within arange between about 60 percent and about 80 percent.
 18. Thecomputer-readable storage media in accordance with claim 15, wherein thecomputer-executable instructions further cause the processor to defineparameters of a flight simulation including at least one of weatherconditions, technological malfunctions, types of aircraft, and thirdparty interference.
 19. The computer-readable storage media inaccordance with claim 15, wherein the computer-executable instructionsfurther cause the processor to adjust the plurality of parameters suchthat the predicted outcome has a quantifiable achievement level of lessthan 100 percent.
 20. The computer-readable storage media in accordancewith claim 15, wherein the computer-executable instructions furthercause the processor to: access actual performance data that includesrecent performance data and older performance data for the at least onelearner determine the predicted outcome for the training program for theat least one learner based on a weighted average of the actualperformance data, the weighted average having a greater emphasis on therecent performance data than the older performance data.